Recommended Frequencies

A recommendation engine for the curation of playlists

Code

%load_ext autoreload
%autoreload 2

Code

from IPython.display import Image
import sys
import os

import catboost
import pandas as pd
import plotly

sys.path.append("../")

Code

# Spotify API interfacte sub-module
from src.spotify.config import (
    MAIN_DATA_FILE, 
    PLAYLIST_FILE, 
    ALBUM_COVER_FILE
)
from src.spotify.utils import read_pickle

# Plotting sub-module
from src.plotting.mood_board import (
    plot_radial_plot, 
    plot_mood_board
)
from src.plotting.album_cover_collage import plot_album_covers

# Streamlit sub-module
from src.streamlit.utils import make_clickable_html

# Modelling sub-module
from src.modelling.config import (
    EUCLIDEAN_FEAT_COLS, 
    CATBOOST_MODEL_FILE
)
from src.modelling.ml_catboost import (
    create_train_test_split,
    get_catboost_predictions,
    train_catboost
)
from src.modelling.ml_data import (
    create_song_pair_features,
    create_song_triplets
)

1 Introduction

«Recommended Frequencies» is a recommendation engine for playlists and currently works for Spotify (the app and its code can be found in this repository). Given a selected playlist in a user’s library, the app suggests songs from the user’s liked songs that may make a good addition to it. The goal of the tool is to provide recommendations solely using information of a user’s library. Thus, methods like collaborative filtering are outside of scope.

Specifically, under the current scope, song suggestions are based on a per-user based Catboost model that uses audio features, song attributes and genre information in the form of embeddings provided by Every Noise at Once. The audio features used by the app are a subset of those provided by Spotify’s API in addition to the year of the song’s album release.

In this blog post I will detail the approach taken to generate music recommendations because afterall:

2 Motivation

Memes aside, the motivation is to improve the listening experience by helping users curate their playlists.

Why playlists you may ask? Well, they are an integral means to enjoying music as 350 million users on Spotify may tell you with their 4 billion playlists. But also just based on the most statistically sound evidence – anecdotal evidence.

Generally, the task of music recommendation is nothing novel. Dubolt, Spotibot.com and even Spotify itself already provide such a service. Specifically, for playlist curation, Spotify provides the Recommended - Based on what's in this playlist option (but it includes songs not in one’s library). Despite this crowded market, I decided to develop a tool myself first and foremost because DIY is fun but also it helped to re-discover my own music.

3 Task

At this point, let me provide a clear description of what I am aiming at achieving.

Imagine having a playlist that captures that feeling of chilling at a pool or lake on a hot summer day. I gave this playlist the very creative name Pool Side Chill (I know 😒 - coming up with good playlist names will be another task to tackle). So I have this concept in my head that the playlist should capture. I also added some songs to it. But I am a bit forgetfull and need a help remembering the songs in my library that fit well into this playlist. This is where Recommended Frequencies comes into play. The goal can be visualised as:

This task of recommending songs for a playlist is multifaceted and difficult since music is subjective. Depending on the intentions of a listener, playlist may capture any of the below concepts or many others:

• Mood (Summertime 🍉)
• Memory (High School 🎓)
• Genre (Rock 🎸)
• Era (80s 📼)

4 Data

In order to tackle this task through modelling, the properties and details of a song need to be quantified in some form, hopefully in way that retains a lot of the multifaceted aspects of music. But it is important to keep in mind that “music’s features do not predict love – music listening does” as the book “This is what it sounds like” by Susan Rogers & Ogi Ogas puts elegantly. So however one attempts to quantify music, it may never really capture the preferences of listeners and what emotion the music evokes.

TODO: Paragraph about extracting attributes of music & how difficult it is…

Luckily, a decent starting point exists in the form of high-level song or rather audio features provided by the Spotify API. The API allows one to pull a lot of information on songs, specifically, I decided to consider the following 11 features of which I have added the official description:

1. Acousticness: A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic.
2. Danceability: Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable.
3. Energy: Energy is a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy.
4. Instrumentalness: Predicts whether a track contains no vocals. “Ooh” and “aah” sounds are treated as instrumental in this context. The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0.
5. Liveness: Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live. A value above 0.8 provides strong likelihood that the track is live.
6. Loudness: The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typically range between -60 and 0 db.
7. Speechiness: Speechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words.
8. Tempo: The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration.
9. Valence: A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).
10. Year of Album Release
11. Genre

Of course many more attributes could be considered, for example the language of the song (if one is, let’s say, curating a playlist of French pop songs). But I am restricting the approach to these for this version of the tool.

These quantitative features of song can be visualised.

For example, let’s take «Cheer Up, My Brother» by HNNY:

Code

# Load saved tracks & their features
df_features = pd.read_pickle(MAIN_DATA_FILE)
# Load all playlists & their songs
list_playlists = read_pickle(PLAYLIST_FILE)
dict_playlists = {playlist["name"][0]: playlist for playlist in list_playlists}

Code

selected_song_id = "37D9O4De2WL1hA6gyResgl"  # Cheer Up, My Brother
selected_song_name, selected_song_artist = df_features.loc[selected_song_id, ["SongName", "Artist"]]

fig = plot_radial_plot(
    df_features.loc[selected_song_id], 
    title=f"Audio features for the song «{selected_song_name}» by {selected_song_artist}",
    inline=False
)
# To render correctly when converting notebook to HMTL using quarto
img_bytes = fig.to_image(format="png", width=600, height=350, scale=5)
Image(img_bytes)

Code

print(f"Genre(s) of {selected_song_artist}:\n\t-  " + "\n\t-  ".join(df_features.loc[selected_song_id, "GenreSet"]))

Genre(s) of HNNY:
    -  swedish house
    -  indie jazz
    -  swedish synthpop
    -  electronica
    -  indie soul
    -  swedish electronic
    -  deep house

5 Methodology

With this quantitative description of music, the challenge is to measure similarity between songs. One important aspect is that similarity of songs should be conditional on the playlist in question. This is because not all attributes of songs should be weighted equally when measuring similarity.

Coming back to the Poll Side Chill playlist for a hot summer day at the pool, the song «Cheer Up, My Brother» by the artist HNNY is a fitting match (it is indeed in my playlist).

Code

selected_playlist = "Pool Side Chill"

playlist_tracks = dict_playlists[selected_playlist]["tracks"]

good_keys = df_features.index.intersection(playlist_tracks)
df_playlist_features = df_features.loc[good_keys]

mood_board = plot_mood_board(
    df_playlist_features[EUCLIDEAN_FEAT_COLS], title="", inline=False, metrics_version=1
)
song_radial_plot_trace = plot_radial_plot(
    df_features.loc[selected_song_id].copy(),
    title=f"{selected_song_name} by {selected_song_artist}",
    only_return_trace=True,
)
mood_board.add_trace(song_radial_plot_trace)
mood_board.update_layout(
    title=f"Song: {selected_song_name} by {selected_song_artist}"
    + "<br>"
    + f"Playlist: {selected_playlist}"
)
# To render correctly when converting notebook to HMTL using quarto
img_bytes = mood_board.to_image(format="png", width=600, height=350, scale=5)
Image(img_bytes)

On the other hand, by the same standards of similarity, the song «Brothers» by City of the Sun seems to be a fitting match for my playlist containing slow paced classical music (which I named «Classico - Adagio»), such as «La Nascita Delle Cose Segrete» by Ludovico Einaudi or «Dream 13 (minus even)» by Max Richter.

But as subjective as music may be, I think we all agree that «Brothers» is not a good match for such a playlist.

Code

selected_playlist = "Classico - Adagio"
selected_song_id = "4lyjuPi3pGN6oTQSK6CUEa"  # Brothers, by City of the Sun
selected_song_name, selected_song_artist = df_features.loc[selected_song_id, ["SongName", "Artist"]]

playlist_tracks = dict_playlists[selected_playlist]["tracks"]

good_keys = df_features.index.intersection(playlist_tracks)
df_playlist_features = df_features.loc[good_keys]

mood_board = plot_mood_board(
    df_playlist_features[EUCLIDEAN_FEAT_COLS], title="", inline=False, metrics_version=1
)
song_radial_plot_trace = plot_radial_plot(
    df_features.loc[selected_song_id].copy(),
    title=f"{selected_song_name} by {selected_song_artist}",
    only_return_trace=True,
)
mood_board.add_trace(song_radial_plot_trace)
mood_board.update_layout(
    title=f"Song: {selected_song_name} by {selected_song_artist}"
    + "<br>"
    + f"Playlist: {selected_playlist}"
)
# To render correctly when converting notebook to HMTL using quarto
img_bytes = mood_board.to_image(format="png", width=600, height=350, scale=5)
Image(img_bytes)

I hope this example illustrates that similarity of songs as described by their high-level audio features is somewhat dependent on the context.

Given this aspect, I decided to solve the task using Machine Learning in order to automatically learn the weights each audio feature should play in the calculation of similarity in the context of a specific playlist. Specifically, a Catboost model is trained to learn to recognise that songs belonging to the same playlist are similar and those from different playlists are contrastive. In other words, the model is trained by setting up positive and negative examples.

This means that triples are created, that are further conditioned on a playlist. Each triple is in the format of
(anchor, positive example, negative example, playlist).

anchor is a specific song and given a playlist (i.e. context), the positive example is another song in the same playlist whereas the negative example is a song in a different playlist. Here, the adjective «different» is important. We want to create clear negative examples because some playlists may be highly related, for example one containing Trap music and the other Hip Hop. I therefore decided to list pairs of similar playlists in my library and only sample negative examples from playlists not deemed similar for a given playlist.

One example of a triple is:

Code

df_triples = create_song_triplets()
example = df_triples.query("playlist == 'Pool Side Chill'").iloc[0]
anchor = "«{0}» by {1}".format(
    *df_features.loc[example.anchor, ["SongName", "Artist"]].values
)
positive_example = "«{0}» by {1}".format(
    *df_features.loc[example.positive_example, ["SongName", "Artist"]].values
)
negative_example = "«{0}» by {1}".format(
    *df_features.loc[example.negative_example, ["SongName", "Artist"]].values
)
print(
    "Anchor: {}\nPositive Example: {}\nNegative Example: {}"
    .format(anchor, positive_example, negative_example)
)

Anchor: «What's the Use?» by Mac Miller
Positive Example: «Cheer Up, My Brother» by HNNY
Negative Example: «Going Through Changes» by Eminem

These triples are then simply transformed into labelled pairs, where the (anchor, positive_example) gets a label of 1 and the pair with the negative_example a label of 0. When constructing the pairs, their feature vectors are concatenated.

For the example above, one gets this feature vector (the suffix _a is for the anchor and _b for the positive example)

Code

df_pairs = create_song_pair_features(df_triples)
(
    df_pairs
    .query("playlist == 'Pool Side Chill'")
    .loc[[(example.anchor, example.positive_example)]]
    .T
)

ID	(2dgrYdgguVZKeCsrVb9XEs, 37D9O4De2WL1hA6gyResgl)
danceability_a	0.759
energy_a	0.492
speechiness_a	0.12
acousticness_a	0.736
instrumentalness_a	0.00989
liveness_a	0.107
valence_a	0.561
tempo_a	0.394704
loudness_a	0.745197
AlbumReleaseYear_a	0.95
GenreEveryNoiseEmbeddingX_a	0.818622
GenreEveryNoiseEmbeddingY_a	0.32913
danceability_b	0.761
energy_b	0.263
speechiness_b	0.0473
acousticness_b	0.12
instrumentalness_b	0.239
liveness_b	0.0996
valence_b	0.315
tempo_b	0.412332
loudness_b	0.632326
AlbumReleaseYear_b	0.9125
GenreEveryNoiseEmbeddingX_b	0.775255
GenreEveryNoiseEmbeddingY_b	0.184596
playlist	Pool Side Chill
target	1

The playlist information is subsequently one-hot-encoded.

With this set-up, the Catboost model is able to pick up on song similarities thanks through the high level audio features but also based on the context (i.e. playlist).

When performing inference, the Catboost model predicts the similarity score (model probability) for every pair between a candidate song and the songs in a playlist. The average similarity scores is then taken for each candidate song. The higher the similarity score, the better the match (according to the model).

6 Results

Having outlined the task and the methodology, we can turn to some results.

6.1 Example # 1

I have decided to create a playlist capturing my favourite music with that 80’s vibe.

Code

album_covers = read_pickle(ALBUM_COVER_FILE)
d_playlist_album_covers = {i[0]: i[-1] for i in album_covers}

Code

selected_playlist = "80's"
fig = plot_album_covers(d_playlist_album_covers[selected_playlist], facecolor="white")

I have already managed to populate it with some songs, here’s an extract:

Code

COL_ORDER = [
    # "PreviewURL", 
    "SongName", 
    "Artist"
]
playlist_tracks = dict_playlists[selected_playlist]["tracks"]
playlist_tracks = list(
    set(dict_playlists[selected_playlist]["tracks"]) &
    set(df_features.index)
)
(
    df_features
    .loc[playlist_tracks]
    .reset_index()[COL_ORDER]
    .head(10)
    # .style.format({"PreviewURL": make_clickable_html})
)

	SongName	Artist
0	Easy Lover	Philip Bailey | Phil Collins
1	Heaven Is a Place On Earth	Belinda Carlisle
2	What About Me	Moving Pictures
3	Karma Chameleon	Culture Club
4	Ti Amo	Umberto Tozzi
5	Take My Breath Away - Love Theme from "Top Gun"	Berlin
6	99 Luftballons	Nena
7	I Promised Myself	Nick Kamen
8	Tainted Love	Soft Cell
9	Sweet Harmony	The Beloved

and now I would like to sift through the rest of my library and get some good recommendations of songs that I could add to this playlist.

Code

# Train the catboost model
# If already trained, load it from disk
if not os.path.exists(CATBOOST_MODEL_FILE):
    df_example_triplets = create_song_triplets()
    df_features_for_model = create_song_pair_features(df_example_triplets)
    df_train, df_test = create_train_test_split(df_features_for_model)
    model_catboost = train_catboost(df_train, df_test)
    model_catboost.save_model(CATBOOST_MODEL_FILE)
else:
    model_catboost = catboost.CatBoostClassifier()
    model_catboost.load_model(fname=CATBOOST_MODEL_FILE)

Code

# Obtain suggestions

# Track information in a pretty format
df_track_info = df_features[[i for i in COL_ORDER if i != "ID"]].copy()

# Songs and their features for modelling (those of the playlist)
df_playlist_features = df_features.loc[playlist_tracks]

# Songs we we consider for predictions and their features
# only considering songs that are not already in the playlist
songs_available_for_suggestion = list(
    set(df_features.index) - 
    set(dict_playlists[selected_playlist]["tracks"])
)
df_songs_available_for_suggestion_features = df_features.loc[
    songs_available_for_suggestion
].copy()

# For each candidate song in `songs_available_for_suggestion`
# measure the similarity w.r.t every song in the playlist
# Sort the candidate songs by decreasing similarity
recommendations = get_catboost_predictions(
    df_playlist_features=df_playlist_features,
    df_songs_available_for_suggestion_features=df_songs_available_for_suggestion_features,
    model_catboost=model_catboost,
    playlist_name=selected_playlist,
)
top_10_recommendations = recommendations.head(10)

# Make results pretty
df_suggested_songs_info = df_track_info.join(
    top_10_recommendations, how="inner"
).sort_values(by="Similarity", ascending=False)

Code

df_suggested_songs_info.reset_index(drop=True)

	SongName	Artist	Similarity
0	Holding Out for a Hero - From "Footloose" Soun...	Bonnie Tyler	0.988707
1	Maniac	Michael Sembello	0.976435
2	Take Me Home - 2016 Remaster	Phil Collins	0.969008
3	West End Girls - 2001 Remaster	Pet Shop Boys	0.967344
4	Alan Watts Blues	Van Morrison	0.961876
5	Conga	Gloria Estefan	0.960995
6	Mercy	The Third Degree | Jon Allen	0.960296
7	I'm Gonna Be (500 Miles)	The Proclaimers	0.959934
8	Footloose - From "Footloose" Soundtrack	Kenny Loggins	0.957814
9	Money's Too Tight (To Mention) - 2008 Remaster	Simply Red	0.956260

Let’s look at how well the first recommendation fits visually when looking at the high level adio/song features.

Code

selected_song_id = df_suggested_songs_info.index[0]
selected_song_name, selected_song_artist = df_features.loc[selected_song_id, ["SongName", "Artist"]]

mood_board = plot_mood_board(
    df_playlist_features[EUCLIDEAN_FEAT_COLS], title="", inline=False, metrics_version=1
)
song_radial_plot_trace = plot_radial_plot(
    df_features.loc[selected_song_id].copy(),
    title=f"{selected_song_name} by {selected_song_artist}",
    only_return_trace=True,
)
mood_board.add_trace(song_radial_plot_trace)
mood_board.update_layout(
    title=f"Song: {selected_song_name} by {selected_song_artist}"
    + "<br>"
    + f"Playlist: {selected_playlist}"
)
# To render correctly when converting notebook to HMTL using quarto
img_bytes = mood_board.to_image(format="png", width=600, height=350, scale=5)
Image(img_bytes)

The model really picked a good candidate here! (That’s my preference here, but maybe someone else disagrees😅).

Let’s have a look at a second example.

6.2 Example # 2

The second playlist I want to showcase is one that contains my favourite rock-ish / folk-ish songs, that are but slow.

Code

selected_playlist = "Rock - Folk"
fig = plot_album_covers(d_playlist_album_covers[selected_playlist], facecolor="white")

The playlist features only few albums/artists, that’s why some of the album covers are duplicated above. So this makes the playlist a great choice, because I would like to diversify it a bit..

Here are some of the songs in the playlist:

Code

playlist_tracks = dict_playlists[selected_playlist]["tracks"]
playlist_tracks = list(
    set(dict_playlists[selected_playlist]["tracks"]) &
    set(df_features.index)
)
(
    df_features
    .loc[playlist_tracks]
    .reset_index()[COL_ORDER]
    .head(10)
)

	SongName	Artist
0	Don't Let Me Be Misunderstood	Nina Simone
1	Mr. Tambourine Man	Bob Dylan
2	Like a Rolling Stone	Bob Dylan
3	Suzanne	Leonard Cohen
4	Hey, That's No Way to Say Goodbye	Leonard Cohen
5	I Wonder	Rodríguez
6	Cause	Rodríguez
7	Crucify Your Mind	Rodríguez
8	So Long, Marianne	Leonard Cohen
9	Hurricane	Bob Dylan

With this information, the model suggests the following Top 10 songs to me:

Code

# Obtain suggestions

# Track information in a pretty format
df_track_info = df_features[[i for i in COL_ORDER if i != "ID"]].copy()

# Songs and their features for modelling (those of the playlist)
df_playlist_features = df_features.loc[playlist_tracks]

# Songs we we consider for predictions and their features
# only considering songs that are not already in the playlist
songs_available_for_suggestion = list(
    set(df_features.index) - 
    set(dict_playlists[selected_playlist]["tracks"])
)
df_songs_available_for_suggestion_features = df_features.loc[
    songs_available_for_suggestion
].copy()

# For each candidate song in `songs_available_for_suggestion`
# measure the similarity w.r.t every song in the playlist
# Sort the candidate songs by decreasing similarity
recommendations = get_catboost_predictions(
    df_playlist_features=df_playlist_features,
    df_songs_available_for_suggestion_features=df_songs_available_for_suggestion_features,
    model_catboost=model_catboost,
    playlist_name=selected_playlist,
)
top_10_recommendations = recommendations.head(10)

# Make results pretty
df_suggested_songs_info = df_track_info.join(
    top_10_recommendations, how="inner"
).sort_values(by="Similarity", ascending=False)

Code

df_suggested_songs_info.reset_index(drop=True)

	SongName	Artist	Similarity
0	Quizas, Quizas, Quizas (Perhaps, Perhaps, Perh...	Nat King Cole	0.981620
1	Blowin' in the Wind	Bob Dylan	0.970832
2	Come Fly With Me	Frank Sinatra	0.968261
3	Beyond the Sea	Bobby Darin	0.967110
4	The Times They Are A-Changin'	Bob Dylan	0.967107
5	Four Women	Nina Simone	0.961116
6	It Ain't Me, Babe	Bob Dylan	0.961099
7	Positively 4th Street	Bob Dylan	0.958995
8	Feeling Good	Nina Simone	0.954881
9	American Pie	Don McLean	0.952283

The model mostly sticked to Bob Dylan and Nina Simone (a safe recommendation!) but it also nicely diversified.

I especially appreciate the suggestion of «American Pie» by Don McLean.

Code

selected_song_id = df_suggested_songs_info.index[-1]
selected_song_name, selected_song_artist = df_features.loc[selected_song_id, ["SongName", "Artist"]]

mood_board = plot_mood_board(
    df_playlist_features[EUCLIDEAN_FEAT_COLS], title="", inline=False, metrics_version=1
)
song_radial_plot_trace = plot_radial_plot(
    df_features.loc[selected_song_id].copy(),
    title=f"{selected_song_name} by {selected_song_artist}",
    only_return_trace=True,
)
mood_board.add_trace(song_radial_plot_trace)
mood_board.update_layout(
    title=f"Song: {selected_song_name} by {selected_song_artist}"
    + "<br>"
    + f"Playlist: {selected_playlist}"
)
# To render correctly when converting notebook to HMTL using quarto
img_bytes = mood_board.to_image(format="png", width=600, height=350, scale=5)
Image(img_bytes)

7 Conclusion

So this has been my solution to personalised music recommendation for curating playlists.

I hope this is applicable and useful to how you organise and consume music and please check out the webapp for yourself. It should be fairly easy to get running on your computer. If have any thoughts or trouble with the code, always happy to hear any feedback:)